the recalibratiNN package
Prof: Guilherme Rodrigues Departament of Statistics/UnB
Low error is not the only asset a NN (or models in general) should have.
It should be able to quantify its uncertainty.
Neural networks can be constructed in a way that yields probabilistic results:
However, like any model , it can be miscalibrated.
Formally:
\[\mathbb{P}(Y \leq \hat{F_Y}^{-1}(p))= p , \forall ~ p \in [0,1]\]
Consider a synthetic data set \((x_i, y_i), i \in (1, ..., n)\) generated by an heteroscedastic non-linear model:
\[ x_i \sim Uniform(1,10)\\ \]
\[ y_i|x_i \sim Normal(\mu = f_1(x_i), \sigma= f_2(x_i)) \\ f_1(x) = 5x^2 + 10 ~; ~ f_2(x) = 30x \]
And the fitted model,
\[ \hat{y}_i = \beta_0 + \beta_1 x_i +\epsilon_i, ~\epsilon_i ~ iid \sim N(0,\sigma) \]
Global Coverage: 94.45%.
Histogram of Probability Integral Transform (PIT) values.
Let \(F_Y(y)\) be the CDF of a continuous random variable Y, then:
\[U = F_Y (Y ) ∼ Uniform(0, 1)\]
\[Y = F_Y^{-1} (U) ∼ Normal(\mu, \sigma)\]
R: probably
Python: ml_insights
Only global, focused on classification problems, and only applicable in the covariate space.
Objective:
Optimize and wrap the code in user-friendly functions:
IDE RStudio, documentation with the package roxygen2 and vignettes/paper in .Rmd.
Package consists of two fundamental components: model diagnostics and model recalibration.
Method:
Package available in this GitHub repository.
The package can be installed and loaded using the following code:
7 functions & 10 dependencies: stats, 6 from the tidyverse , RANN (Approximate KNN), Hmisc (weighted variance), and glue (Customization of plots).
Gaussian models: linear models adjusted by OLS, ANN trained using the least squares MSE loss function.
pit <- PIT_global(ycal = y_cal, # true values from calib. set.
yhat = y_hat_cal, # predictions for calb. set.
mse = MSE_cal) # MSE from calibration set.
head(pit, 5) # observe the first values.[1] 0.04551257 0.42522358 0.81439164 0.69119416 0.44043239
pit_local <- PIT_local(xcal = x_cal,
ycal = y_cal,
yhat = y_hat_cal,
mse = MSE_cal,
clusters = 6,
p_neighbours = 0.2,
PIT = PIT_global)
head(pit_local, 5) # A tibble: 5 × 5
part y_cal y_hat pit n
<glue> <dbl> <dbl> <dbl> <dbl>
1 part_1 -27.9 -8.12 0.456 400
2 part_1 8.63 -8.22 0.538 400
3 part_1 67.4 -7.58 0.663 400
4 part_1 1.58 -7.44 0.520 400
5 part_1 18.4 -8.47 0.560 400
# recalibration
rec_loc <- recalibrate(yhat_new = y_hat_new,
space_new = x_new,
space_cal = x_cal,
pit_values = pit,
mse = MSE_cal,
type = "local",
p_neighbours = 0.2,
epsilon = 0.1)
names(rec_loc)[1] "y_hat_calibrated" "y_var_calibrated"
[3] "y_samples_calibrated_wt" "y_samples_calibrated_raw"
[5] "y_kernel"
Empirical pit distribution.
library(keras)
model <- keras_model_sequential() # simple MLP
model %>%
layer_dense(input_shape = 1, # one covariate
units = 200, # 200 neuron
activation = "sigmoid") %>% # non-linearity
layer_dense(units = 1, # output
activation = "linear") # regression
# compiling model
model %>%
compile(optimizer = optimizer_adam(
learning_rate = 0.01),
loss="mse") # normal distribution
# fitting model
model %>%
fit(x = x_train, y = y_train,
validation_data = list(x_cal, y_cal),
callbacks = callback_early_stopping(
monitor = "val_loss",
patience = 20,
restore_best_weights = T),
batch_size = split*n, # batch size
epochs = 500)
# predicting output values
y_hat_cal <- predict(model, x_cal) # predictions cal
y_hat_new <- predict(model, x_new) # predictions new
# predicition intermediate layer
layer_name <- paste('dense', 1, sep = '_')
layer_model <- keras_model(
inputs = model$input,
outputs = get_layer(model, layer_name)$output
)
h_new <- layer_model %>% predict(x_new)
h_cal <- layer_model %>% predict(x_cal)
# MSE
MSE_cal <-model %>%
evaluate(x_cal, y_cal)
MSE_cal <-as.numeric(MSE_cal)